WIP: OCPNODE-2462 - Split Filesystem OCP Enhancement #1657

kannon92 · 2024-07-31T21:19:10Z

This is a OCP enhancement for enabling KEP-4191 in openshift.

openshift-ci · 2024-07-31T21:19:25Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign ashcrow for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

enhancements/kubelet/split-filesystem.md

rphillips · 2024-07-31T22:31:33Z

enhancements/kubelet/split-filesystem.md

+  - https://github.com/containers/storage/pull/1885
+
+- How does one delete all images and containers once the container runtime config is changed?
+  - crictl on all images and containers on each node?


This could be an out-of-scope item. There is an argument that externally managed storage is owned by the customer.

My main reason for listing this out is what happens if you change from no image store to a new location for image store. Kubelet GC of the existing images would not clean that up.

And tbh crictl wouldn't either. I guess we would require manual deletion if they ran out of disk space.

enhancements/kubelet/split-filesystem.md

rphillips · 2024-07-31T23:13:28Z

Technically, I think we should reserve, IMAGE_STORE as a partition label.
This document could go into this. The script can attempt to mount by label the IMAGE_STORE partition.

The split-filesystem.sh script would:

mount by label the partition named IMAGE_STORE
check to see if the relabelling has been done
- if not, relabel the filesystem
  - touch a file within mounted filesystem on completion of the relabelling (ie: .relabel-complete)

Node Team's scope starts at IMAGE_STORE partition label... This partition label could be setup by CAPI or by documentation or by some other external entity.

haircommander · 2024-08-01T18:14:00Z

enhancements/kubelet/split-filesystem.md

+
+```storage.bu
+variant: openshift
+version: 4.14.0


does this work with 4.17.0? if so we should use the newest version IMO

Interestingly enough I only got this to work with 4.14.

enhancements/kubelet/split-filesystem.md

haircommander · 2024-08-01T18:16:57Z

enhancements/kubelet/split-filesystem.md

+
+Since the image cache has changed locations, all the old images left over should be removed.
+
+Simplest option is to remove the images on each node that this feature was enabled.


should spell out what to do with the containers (which would be left over after the reboot), especially because a lot of those containers will require a running network pod to remove

I have this sorta as an open question.

Do we manually delete all of /var/lib/container/storage on enablement of this feature or disable?

I think using a tool like crictl may actually not work if we change the container runtime. For podman changing these locations does orphan those containers/images.

I think a manual rm may be the best thing we can do.. Unless we delete before enablement of the feature.

Why do we want the old images removed?

enhancements/kubelet/split-filesystem.md

rphillips · 2024-08-01T21:55:30Z

enhancements/kubelet/split-filesystem.md

+
+Since the image cache has changed locations, all the old images left over should be removed.
+
+Simplest option is to remove the images on each node that this feature was enabled.


Why do we want the old images removed?

enhancements/kubelet/split-filesystem.md

rphillips · 2024-08-02T17:37:30Z

We are going to take this enhancement to a meeting to discuss further design.

openshift-bot · 2024-09-03T09:15:25Z

Inactive enhancement proposals go stale after 28d of inactivity.

See https://github.com/openshift/enhancements#life-cycle for details.

Mark the proposal as fresh by commenting /remove-lifecycle stale.
Stale proposals rot after an additional 7d of inactivity and eventually close.
Exclude this proposal from closing by commenting /lifecycle frozen.

If this proposal is safe to close now please do so with /close.

/lifecycle stale

kwilczynski · 2024-09-03T09:59:45Z

/remove-lifecycle stale

enhancements/kubelet/split-filesystem.md

openshift-ci · 2024-09-20T13:25:11Z

@kannon92: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-bot · 2024-10-19T01:15:40Z

Inactive enhancement proposals go stale after 28d of inactivity.

See https://github.com/openshift/enhancements#life-cycle for details.

Mark the proposal as fresh by commenting /remove-lifecycle stale.
Stale proposals rot after an additional 7d of inactivity and eventually close.
Exclude this proposal from closing by commenting /lifecycle frozen.

If this proposal is safe to close now please do so with /close.

/lifecycle stale

openshift-bot · 2024-10-26T08:45:41Z

Stale enhancement proposals rot after 7d of inactivity.

See https://github.com/openshift/enhancements#life-cycle for details.

Mark the proposal as fresh by commenting /remove-lifecycle rotten.
Rotten proposals close after an additional 7d of inactivity.
Exclude this proposal from closing by commenting /lifecycle frozen.

If this proposal is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

openshift-bot · 2024-11-03T00:15:19Z

Rotten enhancement proposals close after 7d of inactivity.

See https://github.com/openshift/enhancements#life-cycle for details.

Reopen the proposal by commenting /reopen.
Mark the proposal as fresh by commenting /remove-lifecycle rotten.
Exclude this proposal from closing again by commenting /lifecycle frozen.

/close

openshift-ci · 2024-11-03T00:15:28Z

@openshift-bot: Closed this PR.

In response to this:

Rotten enhancement proposals close after 7d of inactivity.

See https://github.com/openshift/enhancements#life-cycle for details.

Reopen the proposal by commenting /reopen.
Mark the proposal as fresh by commenting /remove-lifecycle rotten.
Exclude this proposal from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-ci bot requested review from jerpeter1 and runcom July 31, 2024 21:19

kannon92 force-pushed the split-filesystem branch 2 times, most recently from eb11659 to 957070f Compare July 31, 2024 21:36

rphillips suggested changes Jul 31, 2024

View reviewed changes

kannon92 requested a review from rphillips August 1, 2024 16:03

kannon92 force-pushed the split-filesystem branch from 5ce0e41 to 1a2a504 Compare August 1, 2024 16:20

haircommander reviewed Aug 1, 2024

View reviewed changes

enhancements/kubelet/split-filesystem.md Outdated Show resolved Hide resolved

haircommander reviewed Aug 1, 2024

View reviewed changes

enhancements/kubelet/split-filesystem.md Outdated Show resolved Hide resolved

haircommander reviewed Aug 1, 2024

View reviewed changes

enhancements/kubelet/split-filesystem.md Outdated Show resolved Hide resolved

haircommander reviewed Aug 1, 2024

View reviewed changes

enhancements/kubelet/split-filesystem.md Outdated Show resolved Hide resolved

kannon92 force-pushed the split-filesystem branch from 1a2a504 to 31d1b59 Compare August 1, 2024 20:20

OCPNODE-2461: enhancement for split filesystem

3252e2d

kannon92 force-pushed the split-filesystem branch from 31d1b59 to 3252e2d Compare August 1, 2024 20:23

rphillips reviewed Aug 1, 2024

View reviewed changes

kannon92 changed the title ~~OCPNODE-2462 - Split Filesystem OCP Enhancement~~ WIP: OCPNODE-2462 - Split Filesystem OCP Enhancement Aug 6, 2024

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 6, 2024

openshift-ci bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 3, 2024

openshift-ci bot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 3, 2024

wip

bc06232

JoelSpeed reviewed Sep 19, 2024

View reviewed changes

enhancements/kubelet/split-filesystem.md Outdated Show resolved Hide resolved

update with manual walkthrough

33f0ce5

openshift-ci bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Oct 19, 2024

openshift-ci bot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Oct 26, 2024

openshift-ci bot closed this Nov 3, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: OCPNODE-2462 - Split Filesystem OCP Enhancement #1657

WIP: OCPNODE-2462 - Split Filesystem OCP Enhancement #1657

kannon92 commented Jul 31, 2024 •

edited

Loading

openshift-ci bot commented Jul 31, 2024

rphillips Jul 31, 2024

kannon92 Aug 1, 2024

rphillips commented Jul 31, 2024 •

edited

Loading

haircommander Aug 1, 2024

kannon92 Sep 20, 2024

haircommander Aug 1, 2024

kannon92 Aug 1, 2024

rphillips Aug 1, 2024

rphillips Aug 1, 2024

rphillips commented Aug 2, 2024

openshift-bot commented Sep 3, 2024

kwilczynski commented Sep 3, 2024

openshift-ci bot commented Sep 20, 2024

openshift-bot commented Oct 19, 2024

openshift-bot commented Oct 26, 2024

openshift-bot commented Nov 3, 2024

openshift-ci bot commented Nov 3, 2024


		Since the image cache has changed locations, all the old images left over should be removed.

		Simplest option is to remove the images on each node that this feature was enabled.

WIP: OCPNODE-2462 - Split Filesystem OCP Enhancement #1657

WIP: OCPNODE-2462 - Split Filesystem OCP Enhancement #1657

Conversation

kannon92 commented Jul 31, 2024 • edited Loading

openshift-ci bot commented Jul 31, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rphillips commented Jul 31, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rphillips commented Aug 2, 2024

openshift-bot commented Sep 3, 2024

kwilczynski commented Sep 3, 2024

openshift-ci bot commented Sep 20, 2024

openshift-bot commented Oct 19, 2024

openshift-bot commented Oct 26, 2024

openshift-bot commented Nov 3, 2024

openshift-ci bot commented Nov 3, 2024

kannon92 commented Jul 31, 2024 •

edited

Loading

rphillips commented Jul 31, 2024 •

edited

Loading